On the Uniform Sampling of the Web: An Improvement on Bucket Based Sampling

Web is one of the biggest sources of information. The tremendous size, the dynamicity, and the structure of the Web have made the information retrieval process of the Web a challenging issue. Web search engines (WSEs) have started to help users with this matter. However, these types of application, to perform more effectively, always need current information about many characteristics of the Web. To determine these characteristics, one way is to use statistical sampling of the Web pages. In this kind of approaches, instead of analyzing a large number of Web pages, a rather smaller and more uniform set of Web pages is used. This research attempts to analyze the presented methods for generating uniform samples of the pages from the World Wide Web. It specifically focuses on a new method called BBS. Briefly, we improved BBS at least by 4.45% regarding the uniformity of the samples. Using this improved BBS, we estimated the size of the public indexable Web at 27.4 Billion pages. The index sizes of some commercial WSEs are also estimated and compared.